IN THE CLAIMS 



1 . (currently amended) A computer-implemented process for segmenting 
writings written on a whiteboard work surface, comprising the process actions of: 

capturing images of a whiteboard that includes one or more items 
projected on the whiteboard and writings written on the whiteboard with a camera; 

separating into two files the one or more items projected on the 
whiteboard and the writings written on the whiteboard, wherein separating into 
two files the one or more items projected on the whiteboard and the writings 
written on the whiteboard comprises the process actions of: 

determining a relationship between the position of the camera view and the 
one or more items projected on the whiteboard via geometric calibration; 

determining a relationship between the colors of the writings written on the 
whiteboard and the colors of the writings written on the whiteboard captured in 
the images via color calibration: 

estimating a visual echo image that corrects a captured image to match the 
whiteboard in geometry and color using the results of the geometric calibration 
and the color calibration: 

using visual echo cancellation to isolate writings written on the whiteboard by 
comparing the ratio of the albedo of the captured image to the albedo of the 
visual echo image: and 
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displaying the writings written on the whiteboard on another display . 

2. (original) The computer-implemented process of Claim 1 further 
comprising the process action of transmitting the file containing the writings 
written on the whiteboard to remote participants. 

3. (original) The computer-implemented process of Claim 1 further 
comprising the process action of archiving the file containing the writings written 
on the whiteboard. 

4. (original) The computer-implemented process of Claim 1 wherein 
the one or more items projected on the whiteboard comprises an electronic 
presentation. 

5. (original) The computer-implemented process of Claim 1 wherein 
the one or more items projected on the whiteboard comprises annotations 
received from remote participants. 

6. (cancelled) The computer-implemented process of Claim 1 wherein 
separating into two files the one or more items projected on the whiteboard and 
the writings written on the whiteboard comprises the process actions of: 
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determining a relationship between the position of the camera view and the 
one or more items projected on the whiteboard via geometric calibration; 

determining a relationship between the colors of the writings written on the 
whiteboard and the colors of the writings written on the whiteboard captured in 
the images via color calibration; 

estimating a visual echo image that corrects a captured image to match the 
whiteboard in geometry and color using the results of the geometric calibration 
and the color calibration; 

using visual echo cancellation to isolate writings written on the whiteboard by 
comparing the ratio of the albedo of the captured image to the albedo of the 
visual echo image. 

7. (currently amended) The computer-implemented process of 
Claim [[6]] 1 wherein the visual echo comprises annotations from remote 
participant and projected content. 

8. (currently amended) The computer-implemented process of 
Claim [[6]] 1 wherein if the ratio of the albedo of the captured image to the 
albedo of the visual echo is less than 1, then a given pixel[x,y] belongs to the 
writings if and only if 

1 -(albedo ratio for the sum of the red, green and blue color channels)/3 is 
greater than (the sum of the variances of the pixel for the red, green and blue 
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color channels)/(the sum of the visual echoes for the pixel for the red, green and 
blue color channels). 

9. (original) The computer-implemented process of Claim 8 wherein 
for a pixel [x,y] that belongs to the writings, for a given color channel, the color in 
the captured image can be corrected by multiplying its albedo ratio for the given 
channel by 255. 

10. (currently amended) A computer-implemented process for 
identifying content written on a whiteboard comprising the process actions of: 

obtaining geometric calibration data determining the geometric relationship 
between a whiteboard and a view of said whiteboard captured by a camera; 

obtaining color calibration data determining the color relationship between 
the color of writings written on the whiteboard and the color of writings written on 
a whiteboard in an image captured by said camera; 

capturing an image of said whiteboard having projected content and writings 
written on the whiteboard; 

calculating a visual echo of the captured image by correcting the color and 
geometry of the captured image using the geometric calibration data and the 
color calibration data; and- 

using the ratio of the albedo of the captured image to the albedo of the visual 
echo to identify writings on the whiteboard ; and 

displaying the writings on the whiteboard on another display . 
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1 1 . (original) The computer-implemented process of Claim 1 0 further 
comprising the process actions of: 

segmenting out the identified writings; and 

transmitting the segmented writings to a remote site for display. 

12. (original) The computer-implemented process of Claim 10 further 
comprising the process actions of: 

segmenting out the identified writings; and 

saving the segmented writings for later viewing. 

13. (original) The computer-implemented process of Claim 10 wherein 
obtaining geometric calibration data comprises the process actions of: 

projecting a sequence of rectangular patterns onto the whiteboard and 
simultaneously capturing the sequence of rectangular patterns in a sequence of 
images; 

detecting the corners of the rectangular patterns in the sequence of images 
captured with the camera; and 

using the correspondences between the locations of the corners in the 
sequence of images and the corners of the rectangular patterns on the 
whiteboard to estimate a homography that defines the geometric relationship 
between the camera view and the whiteboard. 



14. (original) The computer-implemented process of Claim 13 wherein 
detecting the corners comprises the process actions of: 

converting the images to grayscale images; 

using edge detection to detect edges of the rectangles in the grayscale 
images thereby creating an edge map; 

using a Hough transform to detect straight lines on the edge map; 

fitting a quadrangle for each of the rectangles using the detected straight 
lines; and 

finding the corners of the quadrangle. 

1 5. (original) The computer-implemented process of Claim 1 0 wherein 
obtaining the color calibration data comprises the process actions of: 

quantizing red, green, blue color space into bins; 

for each of the set of quantized colors, 

projecting a known color onto the full whiteboard that is divided into 
sub-regions; 

taking n images of said known color projected on the whiteboard with a 
camera and storing said n images; 

warping each of said n images into whiteboard coordinates using the 
homography found by the geometric calibration; 

calculating the mean and variance of each color across n frames for 
each sub-region; and 
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creating a lookup table of the mean and variance of each sub-region 
for said known color. 

1 6. (original) The computer-implemented process of Claim 1 0 wherein 
if the ratio of the albedo of the captured image to the albedo of the visual echo is 
equal to 1, then nothing is written on the whiteboard. 

17. (original) The computer-implemented process of Claim 1 0 wherein 
if the ratio of the albedo of the captured image to the albedo of the visual echo is 
less than 1 , then there are writings written on the whiteboard. 

1 8. (original) The computer-implemented process of Claim 1 0 wherein 
if the ratio of the albedo of the captured image to the albedo of the visual echo is 
less than 1 , then a given pixel[x,y] belongs to the writings if and only if 

1 -(albedo ratio for the sum of the red, green and blue color channels)/3 is 
greater than (the sum of the variances of the pixel for the red, green and blue 
color channels)/(the sum of the visual echoes for the pixel for the red, green and 
blue color channels). 

19. (original) The computer-implemented process of Claim 18 wherein 
for a pixel [x,y] that belongs to the writings, for a given color channel, the color in 
the captured image can be corrected by multiplying its albedo ratio for the given 
channel by 255. 
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20. (currently amended) A computer-readable medium having 
computer executable instructions for providing contents written on a whiteboard 
to remote meeting participants, said computer executable instructions 
comprising: 

obtaining geometric calibration data determining the geometric relationship 
between a whiteboard and a view of said whiteboard captured by a camera; 

obtaining color calibration data determining the color relationship between 
the color of writings written on the whiteboard and the color of writings written on 
a whiteboard and the color of the writings written on the whiteboard in an image 
captured by said camera; 

capturing an image of said whiteboard having projected content and writings 
written on the whiteboard; 

calculating a visual echo of the captured image by correcting the color and 
geometry of the captured image using the geometric calibration data and the 
color calibration data; and 

displaying the writings written on the whiteboard on another display . 

21 . (currently amended) A system for segmenting a clear image of a 
whiteboard work surface for remote collaboration comprising: 

a camera capturing one or more images of a whiteboard that includes one or 
more items projected on the whiteboard and writings written on the whiteboard; 
and 
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a visual echo cancellation module for separating into two files the one or 
more items projected on the whiteboard and the writings written on the 
wh iteboa rd , wherein the visual echo cancellation module comprises sub-modules 
for: 

obtaining geometric calibration data determining the geometric relationship 
between a whiteboard and a view of said whiteboard captured by a camera; 

obtaining color calibration data determining the color relationship between 
the color of writings written on the whiteboard and the color of the writings written 
on the whiteboard in an image captured by said camera; 

capturing an image of said whiteboard having projected content and writings 
written on the whiteboard; 

calculating a visual echo of the captured image by correcting the color and 
geometry of the captured image using the geometric calibration data and the 
color calibration data; and 

using the ratio of the albedo of the captured image to the albedo of the visual 
echo to identify writings on the whiteboard . 

22. (original) The system of Claim 21 further comprising: 

a remote site with a display for displaying the one or more items 
projected on the whiteboard and writings written on the whiteboard; and 

a network for sending said the one or more items projected on the 
whiteboard and writings written on the whiteboard to said remote site. 

10 



23. (original) The system on Claim 21 wherein the writings written on 
the whiteboard and the one or more items projected on the whiteboard are sent 
to the remote site in two separate files. 

24. (original) The system of Claim 22 wherein the remote site has the 
capability to send back annotations to the whiteboard to be displayed with the 
one or more items projected on the whiteboard. 

25. (cancelledl) 

26. (currently amended) The system of Claim [[25]] 21 wherein if the 
ratio of the albedo of the captured image to the albedo of the visual echo is less 
than 1, then a given pixel[x,y] belongs to the writings if and only if: 

1 -(albedo ratio for the sum of the red, green and blue color channels)/3 is 
greater than (the sum of the variances of the pixel for the red, green and blue 
color channels)/(the sum of the visual echoes for the pixel for the red, green and 
blue color channels). 

27. (original) The system of Claim 26 wherein for a pixel [x,y] that 
belongs to the writings, for a given color channel, the color of the writings in the 
captured image is corrected by multiplying its albedo ratio for the given channel 
by 255. 
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28. (original) The system of Claim 27 wherein for a pixel [x,y] that 
belongs to the writings, for a given color channel, the color in the captured image 
can be corrected by: 

choosing four classes of color corresponding to the most commonly used 
markers used to mark on the whiteboard; 

recovering the set of writings; 

correcting the color of the writings in the captured image for the red, green 
and blue color channels; 

converting the color of the writings from red, green and blue color space to 
hue, saturation and intensity color space to create a new image; 

manually defining writings corresponding to the four classes of color and 
histogramming these writings by hue, saturation and intensity; and 

evaluating the probability the each pixel in the new image belongs to one of 
the four classes based on the histogram. 
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